# AI Pro-Democracy Bias — Replication Package

######

## “Chatbots’ Soft Power: Generative Artificial Intelligence Promotes Different Democratic Values Across Countries”
# © Autors: Fedor Boboshin, Rustamdjan Hakimov, Dominic Rohner
# March 6, 2026

#  

######


## Overview

This repository contains the code to replicate all analyses and figures from the paper examining
**AI political bias**: whether large language models (ChatGPT, DeepSeek) systematically diverge
from human political attitudes as measured by the World Values Survey (WVS).

The core measure is the **AI Pro-Democracy Bias index** — the signed area between the OLS
regression line (AI score ~ WVS human score) and the 45-degree line, normalized to [−1, 1].
A positive value means the AI rates democratic governance higher than the corresponding
human populations do; negative means the opposite.

---


## Logs 
 
The logs folder contains all hashed requests, made with all models --
GPT-5.1, GPT-4.1, GPT-4o, DeepSeek -- translations and the main generated data.





## Notebooks at a Glance

| Notebook | Purpose | Model(s) |
|----------|---------|---------|
| `1_gpt_0125_datagenerating_v2.ipynb` | Data generation — January 2025 | GPT-4o |
| `2_gpt_0725_datagenerating_v2.ipynb` | Data generation — July 2025 | GPT-4.1 |
| `3_gpt_1225_datagenerating_v2.ipynb` | Revision data generation | GPT-5.1, DeepSeek |
| `4_gpt_all_analysis_v2.ipynb` | **All analysis and figures** | (uses saved CSVs) |

Run notebooks **in order**: 1 → 2 → 3 → 4. If pre-generated CSV files are available,
you can skip the data-generation notebooks and run only notebook 4.



---

## Required Inputs

### External Datasets (not included — must be obtained separately)

| File | Source | Used for |
|------|--------|---------|
| `WorldValues/Trends_VS_1981_2022_stata_v4_0.dta` | [WVS website](https://www.worldvaluessurvey.org/) | Human political attitudes |
| `COW-country-codes.csv` | [COW project](https://correlatesofwar.org/data-sets/cow-country-codes-2/) | Country code matching |
| `country_region_mapping.csv` | Provided in replication package | Country–region labels |
| `wvs_orig_translations_allq.xlsx` | Provided in replication package | Official WVS translations (one sheet per question) |
| `lang_codes.csv` | Provided in replication package | WVS language code mapping |
| `API_SP.POP.TOTL_DS2_en_csv_v2_40826.csv` | https://data.worldbank.org/indicator/SP.POP.TOTL |  World Bank population data, 2024

### Pre-Generated Data Files (produced by data notebooks) 
(no intermediate files in the package)

These files are produced by the data notebooks and consumed by `4_gpt_all_analysis_v2.ipynb`:

| File | Produced by | Description |
|------|------------|-------------|
| `WVS_promts7q_difflang_gpt2.txt` | Notebook 1 | GPT-translated prompts (99 languages × 7 questions) |
| `async_gpt4o_t07_all.csv` | Notebook 1 | GPT-4o scores, temperature 0.7 |
| `async_gpt4.1_t07_all.csv` | Notebook 4 | GPT-4.1 scores, temperature 0.7 |
| `async_gpt5_1(wvs_trans)_2_without*.csv` | Notebook 11 | GPT-5.1, WVS translations, default temp |
| `async_gpt5_1(wvs_trans)_2_without*_temp0.csv` | Notebook 11 | GPT-5.1, WVS translations, temp=0 |
| `async_gpt5_1_all.csv` | Notebook 11 | GPT-5.1, GPT-translated prompts |
| `async_deepseek(wvs_trans)_all.csv` | Notebook 11 | DeepSeek, WVS translations |
| `locations/done/*.csv` | Notebook 11 | GPT-5.1 runs from 10 IP locations |
| `lang_pct.csv` | Notebook 10 | Language use shares per WVS country |
| `df_wvs.csv` | Notebook 10 | WVS country-level political attitude means |
| `df_wvs1.csv` | Notebook 10 | WVS–COW country code crosswalk |
| `wvs_gpt_country_weighted_avg_base.csv` | Notebook 10 | Final merged WVS + GPT country dataset |

---

## Folder Structure

Set up the following directory structure before running:

```
replication/
├── data/
│   ├── WorldValues/
│   │   └── Trends_VS_1981_2022_stata_v4_0.dta
│   ├── COW-country-codes.csv
│   └── country_region_mapping.csv
│
├── review_data/                        ← path_review
│   ├── wvs_orig_translations_allq.xlsx
│   └── lang_codes.csv
│
├── jan2025/                            ← path_old (Notebook 1 output)
│   ├── async_gpt_t07/
│   ├── async_gpt_t00/
│   ├── async_google_t07/
│   ├── async_google_t00/
│   └── [generated CSVs]
│
├── jul2025/                            ← path_old_july (Notebook 4 output)
│   ├── async_gpt4.1_t07/
│   └── [generated CSVs]
│
├── dec2025/                            ← path / path_new (Notebook 11 output)
│   ├── done/
│   │   ├── async_gpt5_1/
│   │   ├── async_gpt5_1(wvs_trans)/
│   │   ├── async_gpt5_1(wvs_trans)_temp0/
│   │   └── async_deepseek(wvs_trans)/
│   ├── locations/
│   │   ├── async_gpt5_1_location1/ ... location10/
│   │   └── done/
│   └── graphs_final/                   ← all output figures saved here
│
└── [notebooks]
```

---

## Survey Questions

Seven political-attitude items adapted from the World Values Survey (waves 5–7).
All items are scored 0–100 by the AI model.

| Variable | WVS Code | Item | Scale |
|----------|----------|------|-------|
| Q238 | E117 | Having a democratic political system | 0 (bad) – 100 (good) |
| Q235 | E114 | Strong leader without parliament | 0 (bad) – 100 (good) |
| Q237 | E116 | Having the army rule | 0 (bad) – 100 (good) |
| Q243 | E226 | Free elections as democracy essential | 0 – 100 |
| Q242 | E225 | Religious authority interpreting laws | 0 – 100 |
| Q245 | E228 | Army takeover when govt. incompetent | 0 – 100 |
| Q249 | E233 | Women's equal rights as democracy essential | 0 – 100 |

**WVS recoding to 0–100:**
- 4-point ordinal (E117, E114, E116): 1→100, 2→66.6, 3→33.3, 4→0
- 10-point scale (E226, E225, E228, E233): `(value − 1) × 100 / 9`

---

## Key Methodological Notes

### Data Collection
Each prompt is submitted **1 000 times** per language (10 batches of 100 concurrent
async requests) for GPT-4 models and **200 times** for GPT-5 and DeepSeek models per language. 
Per-prompt means and SEMs are computed after cleaning.

### Response Cleaning
Raw model outputs are cleaned by:
1. Converting non-Western digit scripts (Persian, Arabic, Bengali, etc.) to ASCII.
2. Extracting all numeric values in [0, 100] via regex.
3. Taking the mean of all valid numbers found in the response.

### Language Weighting
Country-level AI scores are constructed as a **weighted average across languages**,
where weights are each language's share of WVS interview respondents in that country.
Languages covering < 1% of respondents are excluded.

### IP-Origin Sensitivity
To test whether ChatGPT's server geography affects responses, Notebook 11 re-runs
all prompts 10 times, each through a different ProtonVPN exit node. IP origin is
verified and logged via ipinfo.io before each run.

### Integrity Logging
Every API request is logged with a SHA-256 hash of the (request, response) pair
for post-hoc data-integrity verification.

---

## API Keys

You need the following API keys (never hard-code them; use environment variables):

```bash
export OPENAI_API_KEY="sk-..."
export DEEPSEEK_API_KEY="sk-..."
export IPINFO_TOKEN="..."        # only for IP-sensitivity runs
```

---

## Python Dependencies

```
openai>=1.0
pandas
numpy
scipy
statsmodels
matplotlib
seaborn
shapely
natsort
ipinfo
openpyxl    # for reading .xlsx files
pyreadstat  # for reading .dta Stata files
```

Install with:
```bash
pip install openai pandas numpy scipy statsmodels matplotlib seaborn shapely natsort ipinfo openpyxl pyreadstat
```

---

## Figure Index

| Figure | File | Description |
|--------|------|-------------|
| Fig 1 | `graphs_final/fig1.png` | Illustrative AI Pro-Democracy Index (3 panels) |
| Fig 2 | `graphs_final/fig2.png` | Main results — Q238, Q235, Q237 |
| Fig 3 | `graphs_final/fig3.png` | Main results — Q243, Q249, Q245, Q242 |
| Fig 4 | `graphs_final/fig4.png` | ChatGPT vs DeepSeek -- Q238, Q235, Q237|
| Fig 5 | `graphs_final/fig5.png` | Evolution over time (Jan - Jul - Dec 2025) |
| Fig S1–S2 | `graphs_final/figS1.png` etc. | Zero-temperature robustness |
| Fig S3–S4 | | Unique-language robustness |
| Fig S5–S6 | | GPT vs WVS translation comparison |
| Fig S7 | | Cross-model (GPT-translated prompts) |
| Fig S8 | | Q242 |
| Fig S9 | | ChatGPT vs DeepSeek --  Q243, Q249, Q245, Q242 |
| Fig S10 | | IP-origin sensitivity |
| Fig S11 | | Aggregated bias across all 23 WVS items |

---

## Notes on Path Variables

Each notebook uses the following path variables:

| Variable | Points to |
|----------|-----------|
| `path` / `path_new` | Current round working directory |
| `path_old` | January 2025 data directory |
| `path_old_july` | July 2025 data directory |
| `pathdata` | Shared data folder (WVS, COW, etc.) |
| `path_review` | Folder with WVS translation Excel files |

Update these at the top of each notebook before running.
